STA 4107/5107 Statistical Learning: Principle Components and Partial Least Squares Regression

Principal components analysis is traditionally presented as an interpretive multivariate technique, where the loadings are chosen to maximally explain the variance in the variable. However, we will consider it here mainly as a statistical learning tool, by using the derived components in a least squares regression to predict unobserved response variables using the principal components. Principal components aims to explain as much of the variation in the data as possible by finding linear combinations that are independent of each other and in the direction of the greatest variation. Each principal component is a linear combination of all variables. The first principal component explains the most variation, the second PC the second most, and so on. There are as many principal components as there are variables, but we usually choose only the first few for both exploratory and regression analysis. Partial least squares is a method of data dimension reduction, similar to principal components, to find the most relevant factors for both prediction and interpretation, and is derived from Herman Wold’s development of iterative fitting of bilinear models (Wold, 1981, 1983). Partial least squares regression (PLSR) improves upon principal components analysis by actively using the response variables during the bilinear decomposition of the predictors. Principal components focuses on the variance in the predictors, while partial least squares focuses on the covariance between the response and the predictors. By balancing the information in both the predictors and the response, PLS reduces the impact of large, but irrelevant predictor variations. Estimation of prediction error is achieved using cross-validation.